1. Field of the Invention
This invention relates generally to the field of measuring and testing and data comprehension and, particularly, to a technique for using sound to identify particular states of a system by storing, manipulating and retrieving data that indicates the status of the system being monitored and using this data as a control source for sound.
2. Description of the Prior Art
A. Introduction
In the fields of measuring and testing and of data comprehension, the primary tools used for user feedback have been visual displays. This includes alphanumeric readouts, dials, indicator lights, computer graphic displays, and so forth. Additionally, auditory feedback has been employed primarily in the form of alarms which sound when certain thresholds are crossed. The purpose of these visual displays has been to provide detailed information about the system being monitored. However, auditory feedback has not been widely employed to provide detailed and continuous information.
As the systems being monitored become increasingly complex, with more individual variables to attend to, a means of integrating the displays may be desirable. This integration allows the system user to make sense of the data he or she is receiving. To this end, color coded meters, 2 and 3 dimensional charts, complex computer displays, and other visual feedback means have been developed (E. Tufte, "Visual Display of Quantitative Information" Connecticut: Cheshire Press, 1983).
In many applications where there are more variables than can be easily integrated into a single visual display, and/or in systems where visual monitoring of displays is not always practical, such as while driving a car or operating machinery, or when the system is being monitored by a vision impaired individual, it may become desirable to use sound as the "display" medium for monitoring the system. We will refer to this use of sound as sonification.
A simple example of sonification might involve controlling a sound's intensity, pitch, harmonic content (brightness), and spatial location to indicate the state of four distinct variables in a system or process being monitored. In order to represent higher dimensions, more complexity is required of the sound. This complexity can be obtained by creating simultaneous auditory streams (polyphony) or by generating a single sound stream with many variables of the sound changing simultaneously.
The use of increasingly complex variables and manipulating variables on different time scales to convey high dimensional information are salient features of sonification. In order to create such complex and multi-variate sounds, the data streams of the system to be displayed auditorially must be translated into a suitable format for controlling a sound generating unit. These formatted control streams are then routed, or `mapped` to selected auditory variables such as pitch, brightness, etc. In this way, a single auditory stream may display multiple data streams.
When a complex auditory stream is used to convey the data, an important perceptual process comes into play. In addition to the system user's ability to scan his or her attention through the sound, relationships between variables and entire system states are perceived `at a glance`. Which is to say, without attention directed effort, all of the auditory variables are perceived as a whole. In addition to a sound being, for example, bright in timbre, high in pitch, pulsing quickly, and loud, it is all at once recognizable as a whole entity.
B. Prior Sonification Work
In the early 1950's Pollack and Ficks (I. Pollack and L. Ficks, "Information of Elementary Multidimensional Auditory Displays", Journal of the Acoustical Society of America, Vol. 26, Number 2, pp. 155-158, March 1954.) published a paper on the use of sound to display data which used a simple binary display technique. They took eight variables and had the test subjects determine whether each variable was in one of two states, e.g. loud or soft, long or short, etc. They concluded that this was an effective technique for conveying data but that "extreme subdivisions of each stimulus dimension does not appear warranted." Later works by E. Yeung (E. S. Yeung, Pattern Recognition by "Audio Representation of Multivariate Analytical Data", Analytical Chemistry, vol. 52, pp. 1120-1123, 1980) and S. Bly (S. Bly, "Sound and Computer Information Presentation", Unpublished dissertation, U. of California, Davis, 1982.) have since explored different techniques, including continuous variation of audible parameters. For an overview of work done to date, the reader is referred to S. Frysinger's "Applied Research in Auditory Data Representation", (Proceedings of the SPIE, E. J. Farrell, Ed; Vol. 1259, pp. 130-139, Bellingham, Wash., 1990). Another example of sonification is the auditory element of "Exvis". a data visualization and sonification software program from the University of Mass. at Lowell.
In a related development, a number of coinposers are using mathematically generated complexity to create compositional forms and/or synthesize sounds. The works of Truax (B. Truax: "Chaotic Nonlinear Systems and Digital Sound Synthesis: An Exploratory Study", Proceedings of the ICMC, Glasgow, 1990), Chareyron (J. Chareyron, "Digital Synthesis of Self-modifying Waveforms by Means of Linear Automata", Computer Music Journal, S. Pope, Ed., vol. 14, #4, MIT Press, 1990., and many others can be cited as examples. The primary difference between sonification and composition as regards embedding information in an audio stream is that in sonification the subsequent extraction of the data for the purposes of understanding the generating system is a primary consideration. In composition this is usually not the case.
In addition to the above cited research, two patents of importance to the present invention are referenced. The first, a patent of W. Kaiser and H. Greiner, ("Warning System for Printing Presses", U.S. Pat. No. 4,224,613), teaches the use of multiple auditory streams to monitor multiple independent data streams. The second, an invention by E. Fubini, A. De Bono, and G. Ruspa, ("System for Monitoring and Indicating Acoustically the Operating Conditions of a Motor Vehicle", U.S. Pat. No. 4,785,280), teaches the use of several parameters of a single auditory stream generated by a sound synthesis system to monitor several data streams.
C. Similar Data Structures for Musical Applications
There are developments in computer music software and hardware that mirror the developments in sonification software. The similarities in file types do not reflect a similarity in function.
In music software, it is common to store data representing musical information such as notes, musical dynamics, tempos, and so on. It is also common in music software and hardware to have files which represent the values of sound parameters which, when retrieved, cause a sound synthesizer to produce a certain timber (sound quality) when played. The musical data file type, commonly found in software `sequencers` (see the "User's Manual for Vision", by Opcode Systems, Menlo Park, Calif.), and the second, commonly accessed via the front panel of music synthesizers as `sound presets` (see "User's Manual for the Korg 01W", Korg U.S.A., Westbury, N.Y.), may roughly correspond to the data component and the sonic map component of auditory beacons, respectively. However, these systems were not designed to be used as described in this disclosure, nor is there any description in any known existing publication of how they may be used to create auditory beacons for data monitoring and comprehension.